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Abstract 

People naturally express themselves through facial gestures and expressions. Our goal 
gesture human-computer interface for use in robot applications. We have implemented a 
a person's facial features in real time (30 Hz). Our system does not require special illumi 
makeup. The work is focused on real-time face tracking using dedicated hardware base 
matching. Tracking using template matching suffers from changing shade and deformati' 
disappearance of facial features. By using multiple Kalman filters we accurately predict e 
facial features. This is despite disturbances and rapid movements of the head (including 
rotational motion). Since we reliably track the face in real-time we are also able to recogr 
of the face. Our system can recognise a large set of gestures ranging from "yes", "no" an' 
detecting winks, blinks and sleeping 
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(chang$3 or alter$4 or modify$4)) 


USPAT 


OR 


OFF 


2003/11/24 14:53 


S52 


0 


(3d near5 (fac$3 near5 template$l)) 
and ((skin or flesh or fac$3) near3 
(chang$3 or alter$4 or modify$4)) 


USPAT 


OR 


OFF 


2003/11/24 14:54 


S53 


0 


((three near2 dimension$3) nearS 
(fac$3 nearS template$l)) and ((skin 
or flesh or fac$3) near3 (chang$3 or 
alter$4 or modify$4)) 


USPAT 


OR 


OFF 


2003/11/24 14:54 


S54 


11 


((three near2 dimens!on$3) near5 
template$l) and ((skin or flesh or 
fac$3) near3 (chang$3 or alter$4 or 
modify$4)) 


USPAT 


OR 


OFF 


2003/11/24 14:56 


S55 


101 


(3d or (three near2 dimension$4)) 
same (fac$3 near3 (template$3 or 
kernal$3 or mask$l)) 


USPAT 


OR 


OFF 


2003/11/24 14:58 
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S56 


290 


template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary) 


USPAT 


OR 


OFF 


2003/11/24 15:07 


S57 


6578 


template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary or 
complementary) 


USPAT 


OR 


OFF 


2003/11/24 15:08 


S58 


5226 


(template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary or 
complementary) ) and fac$3 


USPAT 


OR 


OFF 


2003/11/24 15:08 


S59 


472 


((template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary or 
complementary) ) and fac$3) and 
(fac$3 near3 (determin$6 or locat$6 
or extract$6)) 


USPAT 


OR 


OFF 


2003/11/24 15:09 


S60 


323697 


(((template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary or 
complementary) ) and fac$3) and 
(fac$3 near3 (determin$6 or locat$6 
or extract$6)) ) and face or faces 


USPAT 


OR 


OFF 


2003/11/24 15:10 


S61 


99 


(((template$l same (((skin or flesh) 
near3 color) or (cyan near2 blue) or 
cyanblue or complimentary or 
complementary) ) and fac$3) and 
(fac$3 nearS (determln$6 or locat$6 
or extract$6)) ) and (face or faces) 


USPAT 


OR 


OFF 


2003/11/24 15:10 


S62 


40 


template$ nearS ((different or 
multiple) nearS color$l) 


USPAT 


OR 


OFF 


2003/11/24 15:42 


S63 


2 


"6075557".pn.",6044168".pn. 


USPAT 


OR 


OFF 


2003/11/25 15:56 


S64 


4 


"6044168".pn.",6075557".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IB|v|_TDB 


OR 


OFF 


2004/06/25 15:20 


S68 


4 


"6044168".pn.",6075557".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/03/06 13:50 


S69 


2 


S68 and (user$l or manual$2) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/03/06 13:52 
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S70 


37996 


(((face faces facial) same (rotat$6 
transform$6 mov$6 chang$6 alter$8 
modif$8)) same (user$l or 
manual$2)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:13 


S71 


5397 


S70 and human 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 13:54 


S72 


989 


S70 and ((template$2 kernal$l 
reference$l mask$l) near3 
(compar$4 match$3 correlat$6)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:14 


S73 


966 


S72 and (position$l location$l 
point$l fiducial$l) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:05 


S74 


637 


S73 and imag$4 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBMJTDB 


OR 


OFF 


2005/03/06 14:08 


S75 


197 


S74 and (fac$5 near3 recogni$7) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:09 


S76 


2299 


(((face faces facial) same (rotat$6 
transform$6 mov$6 chang$6 alter$8 
modif$8)) near4 (user$l or 
manual$2)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBMJTDB 


OR 


OFF 


2005/03/06 14:13 


S77 


91 


S76 and ((template$2 kernal$l 
reference$l mask$l) near3 
(compar$4 match$3 correlat$6)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBMJTDB 


OR 


OFF 


2005/03/06 14:35 


S78 


34 


((fac$5 near3 recognition) same 
(template$l kernal$l mask$l and 
refernece$l)) same (user or 
manual$l) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:41 


S79 


231 


((fac$5 near3 recognition) same 
(template$l kernal$l mask$l and 
refernece$l)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 14:41 
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S80 


211 


S79 and (rotat$8 transform$6 alter$9 
modif$7mov$6 chang$5) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^1_TDB 


OR 


OFF 


2005/03/06 14:54 


S81 


167 


S80 and (user$l or manual$3) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/03/06 15:24 


582 


4 


"6044168".pn.",6075557".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IB|V|_TDB 


OR 


OFF 


2005/04/01 11:45 


S83 


309 


(faci$4 near4 (match$4 compar$4 
identif$6)) and ((imag$l same 
(template$l kernal$l mask$l 
reference$l standard$l)) same 
display$2) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 11:50 


S84 


147 


S83 and ((user manual$4) same 
(mov$6 transform$6 rotat$4 alter$4)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IB|V|_TDB 


OR 


OFF 


2005/04/01 12:29 


S85 


247247 


"84" and (face faces facial) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 12:30 


S86 


309 


(faci$4 near4 (match$4 compar$4 
identif$6)) and ((imag$l same 
(template$l kernal$l mask$l 
reference$l standard$l)) same 
display$2) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/04/01 12:29 


S87 


147 


S86 and ((user manual$4) same 
(mov$6 transform$6 rotat$4 alter$4)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 12:29 


S88 


108 


S87 and (face faces facial) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/04/01 15:05 


S89 


5 


'•2324428".pn.",0877274".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM TDB 


OR 


OFF 


2005/04/01 15:09 
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S90 


0 


"00877274".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 15:06 


S91 


5 


"877274".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 15:06 


S92 


4 


"2324428".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 15:09 


S93 


8 


"4996642".pn.",5576778".pn.", 
5983201".pn.",6231188".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/01 15:24 


S94 


11 


((face faces facial) near4 (recogni$6 
identif$8)) same (imag$4 and 
template$l) same (interactiv$5 
manual$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBi^_TDB 


OR 


OFF 


2005/04/02 15:14 


S95 


15 


((face faces facial) near4 (recogni$6 
identlf$8 extract$5)) same (imag$4 
and template$l) same (interactiv$5 
manual$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/02 15:16 


S96 


4 


S95 not S94 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBIM_TDB 


OR 


OFF 


2005/04/02 15:15 


S97 


130 


((face faces facial) near4 (recogni$6 
identif$8 extract$5)) same (imag$4 
and template$l) and (interactiv$5 
manual$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/02 15:16 


S98 


509 


((fac$4 near4 (recogni$7 identif$4 
extract$4)) same template$l) and 
(user$l interactiv$5 manual$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/02 16:53 

> 


S99 


274 


(((face faces facial) near4 (recogni$7 
identif$4 extract$4)) same 
template$l) and (user$l lnteractiv$5 
manual$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBMJTDB 


OR 


OFF 


2005/04/02 16:54 
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SIO 
0 


688 


(fac$4 same template$l) same 
display$4 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEI^; 
IBM_TDB 


OR 


OFF 


2005/04/02 16:54 


SIO 

1 


33 


S99 and (fac$4 same template$l) 
same display$4 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEIVrT; 
IBM_TDB 


OR 


OFF 


2005/04/02 16:54 


SIO 
2 


1 


(((face faces facial) nearS (recogniz$7 
identif$7 extract$5)) same imag$4) 
same (complementary near3 color$l) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEI^; 
IBM_TDB 


OR 


OFF 


2005/04/03 17:32 


SIO 
3 


1 


(((face faces facial) nearS (recogniz$7 
identif$7 extract$5)) same imag$4) 
same (3D and wire) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/04/03 17:34 


SIO 
4 


53 


(((face faces facial) nearS (recogniz$7 
identif$7 extract$5)) same imag$4) 
and (3D and wire) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEITT; 
IBM_TDB 


OR 


OFF 


2005/04/03 17:40 


SIO 
5 


27 


(((face faces facial) nearS (recogniz$7 
identif$7 extract$5)) same imag$4) 
and (3D and wire) and template$ 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEISIT; 
IBM_TDB 


OR 


OFF 


2005/04/03 17:41 


SIO 
6 


0 


(((face faces facial) nearS (recogniz$7 
identif$7 extract$5)) same imag$4) 
and ((3D and wire) same template$) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEI^; 
IBM_TDB 


OR 


OFF 


2005/04/03 17:41 


SIO 
7 


629 


imag$4 and (((face faces facial) near3 
(template$l lcernal$l mask$l)) same 
((face faces facial) near3 (imag$4 
frame$l))) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWEI^; 
IBI^_TDB 


OR 


OFF 


2005/12/27 09:49 


SIO 
8 


290 


S107 and ((transform$3 rotat$4 
mov$4 translat$6) same ((templat$l 
imag$4) and (faces face facial))) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/12/27 10:08 


SIO 
9 


53 


S108 and (((template$l masl<$l 
kernal$l) same (image$l frame$l)) 
same align$5) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^_TDB 


OR 


OFF 


2005/12/27 10:39 
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Sll 
0 


93 


((template$l mask$l kernal$l) same 
(image$l frame$l)) same (face facial 
faces) same (superimpos$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^JTDB 


OR 


OFF 


2005/12/27 11:00 


Sll 

1 


1585 


((template$l mask$l kemal$l) same 
(image$l frame$l)) same (face facial 
faces) same ((on near3 top) above 
layer$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBI^JTDB 


OR 


OFF 


2005/12/27 11:38 


Sll 
2 


213 


Sill and (((extract$4 locat$4 
determin$4) near4 (face faces facial)) 
same imag$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 11:02 


Sll 
3 


67 


((template$l mask$l kernal$l) same 
(image$l frame$l)) same (face facial 
faces) same (overlay $4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 12:13 


Sll 
4 


6673 


((template$l mask$l kernal$l) same 
(image$l frame$l)) same (face facial 
faces) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 12:14 


Sll 
5 


232 


SI 14 and (position$3 near4 match$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 12:14 


Sll 
6 


16 


"6088040" 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 13:47 


Sll 
7 


48 


((extract$6 near4 (face faces facial)) 
near4 (region$l area$l)) same 
((template$l kernal$l mask$l) and 
match$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2005/12/27 13:49 


Sll 
8 


290 


((face faces facial) and (imag$4 and 
(templat$4 kernal$l mask$l))) same 
(superimpos$4 overlay$4 overlaid) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/04 14:52 


Sll 
9 


51 


S118 and (((locat$6 extract$5) near5 
(face faces facial)) same imag$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/04 14:52 
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S12 
0 


727 


((face faces facial) and (imag$4 and 
(templat$4 kernal$l mask$l))) same 
(move moving) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/04 14:52 


S12 
1 


108 


S120 and (((locat$6 extract$5) nearS 
(face faces facial)) same imag$4) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/04 14:52 


S12 
2 


6 


"6088040".pn.",6181806".pn.", 
6035074".pn. 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/06 15:11 


S12 
3 


103 


((extract$4 near4 (face faces facial)) 
near4 imag$4) same template$l same 
((position$l location$l) (match$4 
compar$4)) 


US-PGPUB; 
USPAT; 
EPO; JPO; 
DERWENT; 
IBM_TDB 


OR 


OFF 


2006/01/06 15:29 


S12 
4 


0 


((face nearS recogni$7) same 
(template near5 match$4) same 
(unsharp$5 blurr$6)) 


USPAT 


OR 


OFF 


2006/07/19 16:41 


S12 
5 


30 


((face nearS recogni$7) same 
(template$ nearS match$4)) 


USPAT 


OR 


OFF 


2006/07/19 17:16 


S12 
6 


3 


((unsharp$6 blur$7) nearS (imag$4 
and template$4)) and imag$4 


USPAT 


OR 


OFF 


2006/07/19 17:22 


S12 
7 


14019 


((unsharp$6 blur$7) nearS (innag$4 
template$4)) and imag$4 


USPAT 


OR 


OFF 


2006/07/19 17:23 


S12 
8 


144 


S127 and (tenf)plat$4 near4 nnatch$4) 


USPAT 


OR 


OFF 


2006/07/19 17:41 


S12 
9 


0 


(tenriplate$4 near4 match$4) same 
((unsharp$5 blur$4) near4 
(template$l and imag$4)) 


USPAT 


OR 


OFF 


2006/07/19 17:42 


S13 
0 


8 


(template$4 near4 match$4) same 
((unsharp$5 blur$4) near4 
(template$l imag$4)) 


USPAT 


OR 


OFF 


2006/07/19 17:42 


S13 
1 


0 


(blur$6 near4 template41) and 
(template near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 07:57 


S13 
2 


2 


(blur$6 near4 template$l) and 
(template near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 07:58 


S13 
3 


0 


(unsharp$4 near4 template$l) and 
(template near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 07:59 


S13 
4 


0 


(unsharp$4 near49 template$l) and 
(template near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 07:59 


S13 
5 


7 


(blur$6 near9 template$l) and 
(template near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 08:01 
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S13 
6 


4 


(blur$6 near9 template$l) and 
(position near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 08:01 


S13 
7 


0 


(unsharp$4 near9 template$l) and 
(position near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 08:02 


S13 
8 


4 


(sharp$4 near9 tennplate$l) and 
(position near3 match$4) 


USPAT 


OR 


OFF 


2006/07/20 08:02 


S13 
9 


3 


"6181806".pn.'V6035074".pn.", 
6466685".pn. 


USPAT 


OR 


OFF 


2006/07/20 09:00 


S14 
0 


5251 


382/100, 1 15, 1 184 19, 154, 162, 165, 
181,195,201,209.ccls. 


USPAT 


OR 


OFF 


2006/12/22 10:26 


S14 
1 


555 


S140 and (template$l kernal$l 
mask$l) and (face faces facial) 


US-PGPUB; 

USPAT; 

USOCR; 

EPO; JPO; 

DERWENT; 

IBM_TDB 


OR 


ON 


2006/12/22 10:27 


S14 
2 


20 


S141 and ((unsharp$4 blurr$4 
focus$4) near6 (template$l kernal$l 
mask$l)) 


US-PGPUB; 

USPAT; 

USOCR; 

EPO; JPO; 

DERWENT; 

IBM_TDB 


OR 


ON 


2006/12/22 10:40 


S14 
3 


116 


((face faces facial) and (template$l 
kernal$l mask$l) and (unsharp$5 
biur$4 focus$4)).clm. 


US-PGPUB 


OR 


OFF 


2006/12/22 10:54 


S14 
4 


40 


((face faces facial) and ((template$l 
kernal$l mask$l) same (unsharp$5 
blur$4 focus$4))).clm. 


US-PGPUB 


OR 


OFF 


2006/12/22 10:58 


S14 
5 


76 


S143 not S144 


US-PGPUB; 

USPAT; 

USOCR; 

EPO; JPO; 

DERWENT; 

IBM_TDB 


OR 


ON 


2006/12/22 11:01 
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